Repository navigation
[Auto Parallel] Add spmd rule No.12 for label_smooth and label_smooth ops. - #72845
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
… add_spmd_label_smooth
… add_spmd_label_smooth
ooooo-create
changed the title
label_smooth,expand_as) ops and their grad ops
| } | ||
| SpmdInfo unary_spmd_info = ElementwiseUnaryInferSpmd(label); | ||
| unary_spmd_info.first.push_back(TensorDistAttr()); | ||
| VLOG(4) << "LabelSmoothInferSpmd:"; |
There was a problem hiding this comment.
it is better to put this line before L30/L32 and make the logs more understandable? e.g.
before L30 VLOG(4) << "LabelSmoothInferSpmd call ElementwiseBinaryInferSpmd:"
before L32 VLOG(4) << "LabelSmoothInferSpmd call ElementwiseUnaryInferSpmd:"
| self.assertEqual(inferred_input_dist_attrs[1].dims_mapping, [-1, -1]) | ||
| self.assertEqual(inferred_output_dist_attrs[0].dims_mapping, [0, 1, -1]) | ||
|
|
||
| # shape: [16, 16, 16], [1, 16]. [0, 1, -1], [-1, 0] --> [0, 1, -1], [-1, -1], [0, 1, -1] |
There was a problem hiding this comment.
we need more unit test case to check:
if # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, 1], what result should be
if # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, 0], what result should be
if # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, -1], what result should be
There was a problem hiding this comment.
In the second station # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, 0]
Because of reuse the ElementwiseBinaryInferSpmd, now it will cause error now when meet 1 and 0 with same axes (due to the ShardingMergeForAxis).
The following error is below, and I check the error now rather than implement it
(Unimplemented) Tensor Axis[c] is Sharded by two different mesh dimension [0] and [1]. (at ../paddle/phi/infermeta/spmd_rules/utils.cc:70)
There was a problem hiding this comment.
According to the calculation logic of this operator, if # shape: [16, 16, 16], [1, 16] and dims_mapping: [0, -1, 1], [-1, 0] the result of dims_mapping should be [0, -1, 1], [-1, 1], [0, -1, 1]? if so, we should add some necessary checks and modifications to dims_mapping of prior_dist before calling ElementwiseBinaryInferSpmd?
There was a problem hiding this comment.
Yes. If # shape: [16, 16, 16], [1, 16] and dims_mapping: [0, -1, 1], [-1, 0] the result of dims_mapping should be [0, -1, 1], [-1, 1], [0, -1, 1]
If label's rank is greater than prior_dist's axis. It has an extra_dims_mapping. When prior_dist's dims_mappin in it, replicate it to -1 first. It will work better than before.
(In addition, I think in the future ElementwiseBinaryInferSpmd also should deal with this station
… add_spmd_label_smooth
|
/re-run all-failed |
| check_dim_mapping(forward_info.first[0], {0, 1, -1}); | ||
| const phi::distributed::ArgDistAttr& attr = forward_info.first[1]; | ||
| if (paddle::holds_alternative<phi::distributed::TensorDistAttr>(attr)) { | ||
| EXPECT_EQ(paddle::get<phi::distributed::TensorDistAttr>(attr), |
… ops. (PaddlePaddle#72845) * add spmd_rule for label_smooth * update * fix bugs * apply review * revert due to PaddlePaddle#72753 * revert change for expand_as * Add more tests * apply review * refine code
PR Category
Auto Parallel
PR Types
New features
Description
查看 expand_as 当前已有推导规则实现，故仅仅在 yaml 中进行注册，此外修复了 target_shape 的类型不匹配问题